COMMENT ⊗   VALID 00006 PAGES
C REC  PAGE   DESCRIPTION
C00001 00001
C00002 00002	.SELECT B
C00006 00003	.SELECT B
C00026 00004	.GROUP SKIP 2
C00034 00005	.NEXT PAGE
C00036 00006	.GROUP SKIP 2
C00046 ENDMK
C⊗;
.SELECT B
.BEGIN CENTER
COMPUTER SIMULATION OF MUSIC INSTRUMENT
TONES IN REVERBERANT SPACES
.GROUP SKIP 1
.SELECT 5
Leland C. Smith & John M. Chowning, Department of Music
John M. Grey, Psychology - James A. Moorer, Computer Science - Loren Rush, Music
Consultant - John R. Pierce, California Institute of Technology
.GROUP SKIP 1
Faculty Advisory Group
James B. Angell, Department of Electrical Engineering
John McCarthy, Department of Computer Science
Earl D. Schubert, Hearing & Speech Sciences, School of Medicine
Roger N. Shepard, Department of Psychology
Head of External Advisory Group - Max V. Mathews, Bell Telephone Laboratories
Stanford University, March 1974
.GROUP SKIP 1
.SELECT A
ABSTRACT
.END
.SELECT 1
.BEGIN FILL ADJUST
Novel and powerful computer simulation techniques have been developed which
produce realistic music instrument tones that can be dynamically
moved to arbitrary positions within a simulated reverberant space of
arbitrary size by means of computer control of four loudspeakers.
Research support for the simulation of complex auditory signals
and environments will allow the further development and
application of computer techniques for digital signal processing, graphics,
and computer based subjective scaling, toward the analysis, data reduction,
and synthesis of music instrument tones and reverberant spaces.
Main areas of inquiry are: 1) those physical characteristics of a tone which
have perceptual significance, 2) the simplest data base for perceptual
representation of a tone, 3) the effect of reverberation
and location on the perception of a tone, and 4) optimum artificial
reverberation techniques and position and number of loudspeakers for producing a
full illusion of azimuth, distance, and altitude.  These areas have
been scantily investigated, if at all, and they bear on a larger more
profound problem of intense cross-disciplinary interest: the
cognitive processing and organization of auditory stimuli.
The advanced state of computer technology now makes possible the realization of
a small computer system for the purpose of real-time simulation.
The proposed research includes the specification of, and program
development for, a small special purpose computing system for real-time,
interactive acoustical signal processing.  The research in simulation
and system development has significant applications in a variety
of areas including psychology, education, architectural acoustics,
audio engineering, and music.
.END
.NEXT PAGE
.SELECT B
.GROUP SKIP 1
.BEGIN CENTER
I. INTRODUCTION
.END
.GROUP SKIP 1
.SELECT 1
.BEGIN FILL ADJUST
The program of research  presented here has as its  ultimate goal the
production  of acoustical  waves by  means  of computer  control over
loudspeakers which can provide for  a listener the impression of  any
music instrument tone  at any location within  any reverberant space.
Between  the user who specifies the tone  in a space and the listener
who perceives  it,  there  must be  a  small but  powerful  real-time
computing  system,  a  small  and  highly optimized  data  base,  and
efficient algorithms  which  compute  the  control  signals  for  the
loudspeakers  on  the  basis  of  the   physical  correlates  to  the
perceptual cues of the tone and space.

The  computer production of  acoustical waves which  contain the cues
required for  the %5perception%1  of  a music  instrument tone  in  a
reverberant space is  a problem that is  fundamentally different from
that  of producing waves which  contain all of  the information of an
original instrument source in a real room.  The first we define to be
a problem  of simulation, or the process  of providing the perceptual
impression of  the natural  source,  whereas the  second is  that  of
reproduction, or the process of achieving an exact or close imitation
of the natural waves.  While it is true that an accurate reproduction
of  the  original  acoustical  waves  will  necessarily  contain  the
perceptual  cues of  an  instrument tone  and the  space,  it is  not
necessarily true  that a simulation of a tone will contain all of the
information  in the  original  waves.    We draw  attention  to  this
distinction between reproduction  and simulation, because what can be
learned from  the  two processes  is  significantly different.    The
broadcast, tele-communications, and recording industries have for the
most  part solved  the problems of  reproduction and  the accumulated
knowledge is  vast, having  to  do with  bandwidth, signal  to  noise
ratios, equalization, and encoding.  An equivalent research effort in
simulation  has only just begun, where the  goal is the production of
those  features %5alone%1  of  a  complex  wave to  which  the  human
perceptual mechanisms respond (Risset  & Mathews, 1969).  Research in
the computer  simulation of  complex  auditory signals  will  produce
knowledge  in  the  general  area  of  perceptual  representation  by
isolating  those  physical  features  of  complex  signals which  are
required to give the appearance of naturalness.

It has  become clear  that  for the  purpose of  simulation,  digital
computers provide  the most effective  control of loudspeakers.   The
loudspeaker  is a device  of extraordinary richness  and potential in
that it  can  be  used to  reproduce  nearly any  perceivable  sound,
perhaps  not  perfectly,   but  certainly  with  more  than  adequate
fidelity.   The computer  is  programmed to  generate a  sequence  of
numbers  or samples,  which  are a  numerical  representation of  the
instantaneous amplitude  of a desired waveform.   The accuracy of the
representation increases  as  the  time  interval  between  successive
samples  decreases and  as  the numerical  precision  of each  sample
increases.   The samples are passed in  sequence to digital to analog
converters,  whose  voltage outputs  are  amplified  and  applied  to
loudspeakers.   The precision and flexibility of  this method is very
great and  has  allowed the  development  of analysis  and  synthesis
techniques which are uniquely suitable to digital processing.
.GROUP SKIP 2
%5current research%1

At the  Stanford Artificial  Intelligence Laboratory, pilot  programs
and techniques have  been developed for the analysis and synthesis of
complex signals and  for the  simulation of moving  sound sources  in
reverberant  spaces,   some  of  which  are   startlingly  simple  in
implementation and novel in conception.

Analytical  programs have been written  which digitize the acoustical
wave  of  a music  instrument  by  means  of  an  analog  to  digital
converter.    The  program  then  reduces the  data  to  time-variant
frequency and amplitude functions and the wave is reformed  from
these functions  through additive  synthesis and  played through  the
digital to analog converter in order to determine the goodness of fit
of the analysis.   Further reductions are made  to the data with  the
aim of  discovering the optimal  physical representation of  the tone
with  regard to its  perceptual features. These  procedures have been
successfully applied to tones from several instruments, including the
violin, one  of the most complex  of all instruments,  where the data
reduction ratio is %5greater than 250 to 1!%1 without any disturbance
of the perceptual images of these tones.

An altogether new  technique for generating complex  acoustical waves
using  the  computer was  discovered  here several  years  ago.   The
technique is  based upon a  special application  of simple  frequency
modulation, where with two parameters and two time domain functions a
large  number of  highly differentiated  tones can be  produced which
have a strong resemblance to natural instrument tones.  The technique
does  not have  the generality  of the  additive  synthesis mentioned
above,     however  the   simplicity  of   control  has   provocative
implications.  For many tones the data reduction ratio is a factor of
ten greater than the  ratio for additive synthesis.  It is to a large
degree  the  extraordinarily  simple  physical  correlations  to  the
perceptually complex  images resulting  from this technique  that has
generated   a  far-reaching  research  interest  in  the  significant
perceptual cues for such images.

In the  simulation of  natural  tones in  natural environments  using
loudspeakers,  it is  of utmost  importance that  the realism  of the
auditory images supercedes the physical presence of the  loudspeakers
themselves.   In order  to free  sounds from  the loudspeakers it  is
necessary  to simulate the  reverberation of a  space as  well as the
localization cues of  the  sound within  the  space.   To  this  end,
artificial reverberation  techniques based on Schroeder (1962), together with
the results  of several years of research into producing localization
cues by  means of loudspeakers,  have been  implemented in a  general
control program  for the arbitrary  localization of a  source.  Using
interactive graphic  display  techniques,  a  user  can  specify  the
location an  movement (trajectory)  of a  sound in a  two-dimensional
reverberant  space.   A program  computes  the control  functions for
azimuth, distance, and velocity which are used to modulate the signal
to be applied  to the loudspeakers.  This program  has been useful in
the  investigation of simulation algorithms for localization cues and
for indicating the most potentially productive research areas for the
future.

All of the research to date has been  done at the Stanford Artificial
Intelligence  Laboratory.  It should  be noted that  we have received
%5no direct support for research  in the form of salaries or purchase
of  hardware,%1 although  we have  been allowed  to use  the computer
facilities  through the generosity of its  directors.  Lately, as the
use of these facilities has  increased by a significant amount,   and
consequently,   the presence  of any research  group has been  a much
more apparent  load on  the system,   the  presence of  a  non-funded
project has become an increasing burden on  the resources of the lab.
It has become  clear,  therefore,  that we must seek external support
in  order  to  continue  our  association  with  the  lab,     by  1)
significantly reducing  our computing load  and 2) paying,   in part,
for  our use  of  peripheral equipment.  Recent advances  in computer
technology resulting in the availability of  specialized hardware for
real-time  signal processing,   make  possible our  development  of a
small satellite system which would not only significantly reduce  our
load on the system,  but would immeasurably increase the  rate of our
research progress and  allow for many diverse and unforeseen real-time
applications, some  of which  are  listed below.   The  research  and
development  of such  a system  is a  major impetus  for our  seeking
external support.
.NEXT PAGE
%5proposed research%1

The results from the current and past research indicate clearly both 
the general direction and some of the specific steps for the future.
The overall goal in the simulation of instrument tones is the development
of synthesis algorithms which produce tones having the perceptual
complexity and naturalness of those in the real world, but which also
have the simplest possible physical representation in the computer.
In order to achieve this goal, the analysis and two synthesis techniques
will be applied to a larger set of tones with a view to capturing or confirming
the significant perceptual features through the rigorous application of
reduction techniques.  As the empirical data accumulates, the reduction
techniques will be `formalized' as algorithms which are able to detect
and preserve the perceptual features of a tone in the most concise representation.
Similarly, algorithms for mapping the perceptual attributes of a tone into
parameters and functions for FM synthesis will be developed.  A convergence
of the two synthesis techniques is anticipated in that the FM synthesis
of many complex instrument tones requires a particular expansion of the
technique which places it in part in the category of additive synthesis.
Finally, the perceptual representations of tones will be used to formulate
higher order algorithms which reflect a general model for the perception
of a wide range of natural tones.
Methods from experimental psychology will be used to help
establish the dimensionality of this perceptual model and the 
relationships between the subjective dimensions and the physical properties
of the tones.

In the case of the simulation of reverberant spaces and the localization of
sources within the space, the overall goal is to be able to provide for a 
user the maximum control over localization of sources and over size,
shape, and reverberant qualities of the apparent space.  The major research
to be done in achieving natural representation of real rooms is in the
artificial reverberation algorithms.  We plan to develop the techniques
of Schroeder in conjunction with other techniques developed here, through
the use of graphic computer analysis programs.  The difficulty in constructing
compound reverberation circuits is that there is no current method for
formal prediction of their output (which in addition is very often
counter-intuitive).  The combination of such programs, together with subjective
evaluation, appears to be the most effective manner of research.  With the
application of resonators to uncolored synthesized reverberation, we plan
to simulate a number of real acoustical environments.

The research in localization will focus on the optimum number and arrangement
of independently controlled loudspeakers which maximize the effective area of 
listening positions.  Although the algorithms which
we have developed for localization appear to be effective for
four channels, we plan to further "tune" and evaluate the cues through
subjective measurements for as many as eight channels.

The effectiveness of the research effort will be dependent upon the proposed
development of a special purpose, interactive, acoustical signal
processing system.  The system will be able to synthesize in real-time
a number of complex signals, localized in complex reverberant environments.
Programs will be developed which will give the user a high level control
over the total acoustical environment.  The programs will include
all of the digital simulation techniques which have been developed and which
are proposed.
The integrated circuit technologies, especially the rapidly developing field of
large scale integration, suggest provocative applications of this
research in digital simulation.

.END
.GROUP SKIP 2
.SELECT C
APPLICATIONS
.SELECT 1
.GROUP SKIP 1
.SELECT 5
scientific research
.SELECT 1
.BEGIN FILL ADJUST
Our research and  development of powerful simulation  algorithms will
make  significant contributions to several  related scientific areas,
both of a theoretical and practical nature. The most obvious benefits
which will  directly result from  our research will  be contributions
to,   and implications for,  auditory theory.  Psychoacoustics is now
only beginning  to include the  study of  the perception of  auditory
signals  which resemble  sounds  from our  daily  environments.   The
largest body of research in  this domain has included the  perception
of  speech signals,  an  interest  which  has many  obvious  payoffs.
Little work has been done on the perception of the non-verbal signals
which  constitute  a  sizeable  remainder  of  our  complex,  natural
auditory  environment.   The few  efforts which  have been  made have
attempted  to look at  the perception  of music instrument  tones,  a
logical starting  point for  such investigation.   We  feel that  the
research  which   we  will  undertake  will   make  many  significant
contributions to this  growing area  of study,   an  area which  will
provide  an  ultimate  test for  many  general  models  for  auditory
perception.   Indeed,  implications should result  from our  work for
models of speech perception, in that we are dealing with a comparably
complex auditory domain.

Another contribution of our proposed research to the study of hearing
is in our development of a real-time digital system for the synthesis
of sound.   It would be  conservative to note  that over half of  the
time  spent in  research on  auditory perception  is consumed  by the
construction and debugging of  special-purpose analog circuits.   The
problems with the stability of, and precision of control over, analog
hardware  has imposed implicit  limitations on the  complexity of the
auditory stimuli which can serve as tools in research. It is for this
reason that the study of the  perception of complex,  natural signals
is  out of the reach  of most researchers;  with digital synthesis of
sound it becomes possible,   but with the development of  a real-time
digital system for synthesis it becomes practical.  We feel that both
the research  and development  of a  real-time system,  which  indeed
supports interactive psychoacoustical experimentation,  and the model
which  our research  will  provide to  this branch  of  science, will
represent  a  significant  step  in  research  possibilities  to  the
scientific   community.     For   example,   in  auditory   research,
special-purpose, low  budget systems could evolve from our work which
certainly  would  represent as  much  of  a  jump  from  the  current
limitations of equipment as was the jump from the use of tuning forks
and Helmholtz resonators  to the  use of  electronic oscillators  and
filters.

The algorithms which  we are developing  for the simulation  of music
instrument   tones  localized  in  reverberant   spaces  are  clearly
applicable  to  the  study   of  higher-order  auditory   information
processing.  An extension of the range of auditory signals which
fall  in the domain  of our simulation techniques  would include many
diverse naturalistic  sounds that are  not traditionally  categorized
with   music  instruments,     but  which   occur  in   our  everyday
environments, such as various types of noise, mechanical sounds, and
any of the many other sorts of non-verbal sounds which daily surround us.
Techniques already  implemented for  the simulation  of
localized sources  of  sound in  reverberant spaces  would of  course
apply to the extended  set of auditory signals.  At this point,  very
powerful tools  would exist  for any  social  science research  which
desires to examine the behavior of man in a naturalistic environment,
but  further demands  the  control over  environmental factors.   The
mounting interest  in  this  level of  work  is demonstrated  by  the
increasing use of the criterion of the relevance of research findings
to real-life situations.  The social psychologist could study the 
influences of various auditory conditions on human behavior, such as
levels and types of noise.  The  study of the internal  representations
of naturalistic  auditory signals and  the cognitive  operations which
may be performed on these representations could make many uses of the
tools which we are developing.   Both short and long term memory  for
non-verbal,   but  familiar,     auditory   stimuli  could   also  be
investigated with  these simulation techniques.  They would also make
feasible the controlled investigation of the effects  of training and
experience on the processing of, or the effects of the contexts which
surround,  natural  signals.    These  are  but  a  few  examples  of
applications  for  the  simulation algorithms  which  we  propose  to
develop.
.END
.NEXT PAGE
%5education%1
.BEGIN FILL ADJUST
In music education, hardware implementations of the simulation algorithms
can be coupled with small computer-aided instruction systems.  Placed in a
primary school environment
with keyboard and/or alternative controls, i.e. knobs, joy-sticks, switches,
children could readily explore the basic musical parameters of rhythm and
pitch in the rich context of the orchestral instrument timbres.  The
system could be defined in such a way that basic principles of acoustics
and perception could be simply presented and experimented with by young
children as a way of understanding the physical medium on which the art
of music depends.  At higher levels of education in the high school
and university, the system
would allow a student to actively experiment with principles of orchestration
without concern for the inhibiting cost and/or time of live musicians.  Another more
obvious application is the simulation of tones for computer-aided ear-training strategies
which reflect the complexity and richness of the tones which form the students
natural musical environment, for which his `ears' are being trained.
.END
.GROUP SKIP 2
%5architectural acoustics%1
.BEGIN FILL ADJUST
Architectural acoustics, as has often been stated, lies somewhere between
science and art.  The number and complexity of inter-relationships of the
variables in the acoustics of rooms is enormous.  Any information having
to do with the subjective evaluation of room information is therefore
bound to be useful in the design and construction of acoustical spaces.
The research we propose should be of obvious benefit.

There is another application of our research which may contribute to a
new and revolutionary approach to the entire question of performance
spaces, that is totally artificial reverberation.  The design and construction
of auditoriums is enormously expensive and once completed difficult to alter
according to subjective evaluation.  Unlike current techniques
for the artificial enhancement of natural reverberation, totally artificial
reverberation would eliminate any concern for room shape and size as it
affects reverberation time, frequency response and resonances, flutter,
and first delays.  The only requirement for construction would be
that the room be reasonably dead.  Microphones placed in the room would
pickup the direct signals from the sources and pass them to digital delay
and resonance circuits and then back into the room by means of a number
of loudspeakers arranged in the walls and ceiling.  The apparent room
size and reverberant characteristics, for example concert hall or cathedral,
would be under the control of the musical director and could, in fact, be
changed in the same concert according to the requirements of the program.
In this manner, a rectangular room, relatively inexpensive to build,
would be a flexible performance space, capable of meeting the
acoustical requirements for types of music ranging from chamber music to
large choral and orchestral works.
.END
.GROUP SKIP 2
%5music industry%1
.BEGIN FILL ADJUST
Simulated reverberant spaces and control of source location also
have obvious applications in the cinema and recording industries.  Typically,
music is recorded in ideal studio conditions using a large number of microphones
and recording channels.  In the process of mixing down to two or four channels,
the signals are passed through a reverberation chamber or an electronic
reverberation device to compensate for the dryness of the studio.  Digital
spatial processing systems which allow precise control of reverberation time,
frequency response, and room shape would add a flexibility
to the processing which would not only allow simulated reverberation of a
variety of concert halls, but would also allow localization in distance and
angle of each of the original source channels.

The electronic organ and synthesizer industries are altogether dependent on
simulation techniques and are beginning to utilize digital synthesis,
albeit primitive from the point of view of perception.  The difficulty is
that practical manufacturing costs are incompatible with sensory pleasure and
realism using the current techniques for simulation.  What is required are simplified,
yet effective, perceptual models for additive and subtractive synthesis,
and/or alternative techniques for simulation of pleasing tones, two areas
where our research has great potential.
.END
.GROUP SKIP 2
%5electronic music composition
.SELECT 1
.BEGIN FILL ADJUST
Another high order human activity is music composition.
Here, there is a strong commitment
to the use of the loudspeaker for simulation as well as for amplification.
There is no doubt that with the development of electronic music the
loudspeaker has become the musical instrument of the present and future.
The desire of the composer to expand and control his material is unbounded and it
was such a quest which led to the development of electronic music.
Studios in Europe and America have proliferated as analog synthesizers
have become ever more sophisticated.  The deficiency of analog synthesis, however,
is that it is not sufficiently general to allow the composer to control
his material from the elemental level of the sound itself to the higher levels
of form.  The hybrid system, a computer controlled analog synthesizer, is a solution
to the control on a formal level, but it will not allow the composer to
`get his hands on' the structure of the sound itself.  Digital synthesis
techniques are now seen to be the general solution for the composition of
electronic music - at least they can theoretically produce any perceivable sound.
The remaining difficulty is that very little is known about sound
that is of sufficient complexity (interest) to be useful to composers:
which is to say, sound no less complex than that of musical instruments.

The applications of the proposed research and computer facility to the field
of electronic music are most appropriate.  Realistic simulation
of the natural music instrument tones forms an extraordinarily
rich point of departure for the discovery and manipulation of timbres,
both known and unknown.  The simulation techniques for localization
and reverberant spaces not only liberate the sounds from the loudspeaker,
but allow specification of the `performance room' itself, thereby giving the composer
a control over his medium never before possible.

There is, perhaps, a more profound relationship of the research we propose to
contemporary composition.  Although psychoacoustics is not the principle thrust
of our research, it is a major component of every aspect of our work in the
development of simulation algorithms.  The following quote seems
particularly appropriate.

.SELECT C
.NARROW 10,10
A few years ago, a colleague and I were asked to write a chapter on acoustics
for a book on contemporary music.  We assembled what seemed to us pertinent
current and unexploited information.  The editor rejected the chapter on the
grounds that we had not related the material to examples of contemporary
music.  The only response we could make was that no relation was possible.
Contemporary music and psychoacoustics had become completely disjoint fields.
	- J.R. Pierce
.WIDEN
.END
.NEXT PAGE